fix(backup): stop shipping the archives back through the log pipeline (#286) - #315
Conversation
…#286) `backup-volumes.sh` archives each volume with `docker run ... tar -czf -`, so the gzip stream IS that container's stdout. The json-file driver records stdout whether or not anyone reads it, and Alloy's `loki.source.docker` tails every container on the socket — so `make backup` sent its own output back into Loki as log lines. Measured on 2026-08-29: 765 MB across three volumes in about three minutes, 2.7 MiB/s against a 950 B/s baseline, 5,500 lines a second against three. Loki's RSS tracked it minute for minute up to 1015 MiB, which is what set its `mem_limit` in #114. Roughly three days of the estate's real logs per run, and it recurred on the backup timers — 577 MiB on 09-04. It also quietly undid `archive_one()`'s own argument. That function encrypts with age precisely so the plaintext never touches this disk; the *compressed* plaintext was landing in `loki-data`, which is not encrypted, and staying for the 30-day retention. Fixed at source: `--log-driver none` on the runs whose stdout is a data stream, so Docker discards it rather than Alloy filtering it later. Nothing reads `docker logs` on a container the shell is already piping. Second line of defence in `docker.alloy`: containers labelled `homelab.logs=off` are dropped before Alloy opens a log stream, for the next ad-hoc `docker run` that forgets. NOT filtered on the compose project, which #286 proposed and which would have broken collection. That label is empty for every container on `oracle` — `wiki`, `db` and the agent are all plain `docker run` (scripts/deploy-agent.sh) — so "drop what has no project label" would have silently stopped collecting logs for the one service in this estate anybody uses. Checked against `container_last_seen` on both hosts first. Opt-out is weaker than the allow-list #286 wanted and the gap is written down rather than papered over: a throwaway container setting neither flag is still tailed. Bounding that is a per-stream ingestion limit in Loki, which trades a flood for silently dropped lines and is a separate decision. `loki`'s `mem_limit` is deliberately unchanged. Every measurement behind it still includes the flood; a fortnight without one does not exist until 2026-09-18. The comment says so and says what to re-derive from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deployed and verified
Both directions, because "no flood" and "collection is broken" look identicalThat is the whole risk in this change, and it is the #62 shape: a filter that Read out of And in Loki, with both probes emitting every 20s:
The unlabelled container is still collected. Only the labelled one is dropped. A real
|
| before | after | |
|---|---|---|
| archiver bytes into Loki per run | 765 MB | 0 |
| peak ingest rate | 2769 KiB/s | 77 KiB/s |
| loki peak RSS | 1015 MiB | 229 MiB |
0 OOM kills; all seven services back up with their limits intact.
One number reported straight
Loki's RSS is sitting at 226-229 MiB, not the 107-128 MiB of the quiet days in
the #114 window. That is 4.4x below the flood and 15% of its 1536M ceiling, so
nothing here is at risk — but it is not "within the routine range" that #286's
done-when asked for, and it would be easy to write it up as though it were. The
likely cause is the post-quiesce restart rather than the backup: the stack was
stopped for 107s and Loki replays its WAL and reopens its index on the way back.
It is worth watching rather than explaining away, and the 2026-09-18
re-derivation of mem_limit will settle it either way — if the steady state is
really ~226 MiB rather than ~120, the right ceiling is nearer 768M than 512M and
the comment in compose.yaml should say so.
Still open, by design
loki'smem_limitstays1536Muntil 2026-09-18. Sizing it today would
mean sizing from a window that contains the flood.- No Loki per-stream ingestion limit. That is what would bound a throwaway
container which sets neither flag, and it trades a flood for silently dropped
lines — its own decision, named indocker.alloyand not taken here.
1 participant
Closes #286.
backup-volumes.sharchives each volume withdocker run ... tar -czf -, sothe gzip stream is that container's stdout. The json-file driver records
stdout whether or not anyone reads it, and Alloy's
loki.source.dockertailsevery container on the socket — so
make backupsent its own archives backinto Loki as log lines.
It also quietly undid
archive_one()'s own argument: that function encryptswith
ageprecisely so the plaintext never touches this disk, and thecompressed plaintext was landing in
loki-data, which is not encrypted, forthe 30-day retention.
The fix #286 proposed would have broken log collection
The issue said to relabel on
__meta_docker_container_label_com_docker_compose_projectand drop what doesnot carry it. I checked that against
container_last_seenon both hosts beforewriting any of it:
Everything on
oracleis a plaindocker run(scripts/deploy-agent.sh), sothat filter would have silently stopped collecting logs for the one service
in this estate anybody uses — the #62 shape exactly: no error, no failed
component, just an empty dashboard. An allow-list of known-good names is the
same trap one step further on; it fails closed on a host nobody remembered.
What landed instead
--log-driver noneon the runs whose stdout is a data stream. Dockerdiscards it at source — nothing to tail, nothing to filter. Nothing reads
docker logson a container the shell is already piping.homelab.logs=off, dropped indiscovery.relabelbefore Alloy opens a stream. Second line of defence for the next ad-hoc
docker runthat forgets.Applied to the archiver runs in
backup-volumes.shandrestore-volumes.sh,including the pre-restore safety snapshot, which is the same tar-to-stdout
shape and would have done the same thing.
What this does not fix
Opt-out is weaker than the allow-list #286 wanted, and the gap is in the config
comment rather than papered over: a future throwaway container setting neither
flag is still tailed. What bounds that is a per-stream ingestion limit in
Loki (
limits_configsets none today, so the 2.7 MiB/s flood sailed under the4 MB/s default). That trades a flood for silently dropped lines, which is its
own decision and not this PR's.
loki's mem_limit is deliberately unchanged
#114 sized it at
1536Mfrom a 1015 MiB peak that was this bug. Everymeasurement behind that number still includes the flood, and a fortnight
without one does not exist until 2026-09-18. The comment in
compose.yamlnow carries that date and what to re-derive from — the nine quiet days in the
window peaked at 107-128 MiB, so expect roughly
512M.Moving it now would be sizing from a window I have already established is
contaminated, which is the mistake #114 spent two weeks not making.
Verification
make validatepasses. Post-merge I will deploy and check, positively in bothdirections, that an unlabelled container is still collected and a labelled one
is not — then run a real
make backupand measure. Results in a comment.🤖 Generated with Claude Code